Thanks for the PR! I really like this and I think we should implement this, before we move to DDP (which is the next thing on my todo list), so I just had a look at this PR.
I have two suggestions/adjustments:

1. I would get rid of the example, since it only showcases one changed parameter. We can simply update the documentation.
2. I think the functionality of parameters.running.num_gpus = 4 can simply be absorbed into parameters.use_gpu. That value could simply be an int instead of a bool, without any drawbacks.

I have made these changes in a PR here: nerkulec#1, if that looks OK we could first merge that PR and then this one here.

Oh wait, there is one potential problem, and that is when training with multiple GPUs and then loading to run with either only one GPU or MPI+GPU. I will test this right away!

I like your changes :)
I added a small fix to my part. Now the checks make much more sense. Feel free to merge when you resolve the potential problem you mentioned.

I confirmed that inference pipeline indeed still works!

In theory this works, once this is benchmarked we can merge it!

The benchmarks showed that training on multi-gpu using DataParallel was slower than with one gpu. I'm closing this since we have the DDP implementation.